Facing the challenge in evaluating technology use in mobile environments

Developing innovative mobile approaches to informal and formal learning raises a challenge to satisfy various stakeholders. Evaluations may focus on examining the nature and quality of learning that occurs when using technological systems, other methods are targeted towards user interactions with the systems. In this paper we highlight a methodology that attempts to address these two analytical issues in parallel, and communicate the results across stakeholders. The methodology is grounded in cultural historical activity theory and is compatible with other views emerging of such evaluation as having multiple levels. The method applies task analysis to examine the conflicts that emerge when learners are interacting with technological systems in an informal learning setting. Results from a trial involving first aiders are used to illustrate the techniques as they were applied as part of a European project that developed a collaborative mobile learning environment and results from. The method has been repeated in other studies and can provide a tool to help reflect on understanding and enable the sharing of perspectives on evaluation outcomes.